SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. each distillation iteration, SKD samples a teacher from pre-defined team, which consists of multiple models with multi-level capacities, transfer knowledge into student in an one-to-one manner. Sampling distribution plays important role SKD. We heuristically present three types sampling distributions assign appropriate probabilities for models. has two advantages: 1) it can preserve the diversities via stochastically single and 2) also improve efficacy when large capacity gap exists between model. Experimental results on GLUE benchmark show that SKDBERT reduces size BERT by 40% while retaining 99.5% performances understanding being 100% faster.